Preconditioning via Diagonal Scaling

Interior point methods solve small to medium sized problems to high accuracy in a reasonable amount of time. However, for larger problems as well as stochastic problems, one needs to use first-order methods such as stochastic gradient descent (SGD), the alternating direction method of multipliers (ADMM), and conjugate gradient (CG) in order to attain a modest accuracy in a reasonable number of iterations. The condition number of a matrix A, denoted by κ(A), is defined as the ratio of its maximum singular value to its minimum singular value. Both theoretical analysis and practical evidence suggest that the precision and convergence rate of the first-order methods can depend significantly on the condition number of the matrices involved in the problems. As an example, the CG algorithm for solving the linear system Ax = b achieves a faster rate of convergence when the condition number of A is smaller. Hence, it is desirable to decrease the condition number of matrix A by applying a transformation to it; this process is called preconditioning. A special case of preconditioning is called diagonal scaling. Here, we are interested in finding diagonal matrices D and E to minimize the condition number of the matrix A = DAE, in order to accelerate first-order methods. For example, in applying CG to solve Ax = b, we can solve Ax̃ = Db instead, and recover x = Ex̃, while taking advantage of the small condition number of A. In our numerical experiments and from theoretical analysis, we have concluded that preconditioning can improve the performance of the first-order methods in two different ways. First, it can significantly accelerate the linear algebra operations. For example in [OCPB13], each step of the algorithm involves running CG which can be done remarkably faster if the appropriate preconditioning is applied. The second effect of the preconditioning, which should be distinguished from the first one, is decreasing the number of iterations for achieving desired accuracy by following different intermediate points in ADMM. In this report, we first discuss heuristics for diagonal scaling. Next, we motivate preconditioning by an example, and then we study preconditioning for a specific splitting form in ADMM called graph projection splitting. Finally we examine the performance of our methods by some numerical examples.